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Abstract 

Linear regression examines the relationship between one or more independent (predictor) 
variables and a dependent variable. By using the formula y = a + biXi + b 2 X 2 + e, regression 
determines the weights needed to minimize the error term (e) for a given set of predictors. With 
one predictor variable, the relationship between the predictor and the dependent variable is 
linear. With two predictors, this relationship becomes planar, and with three or more predictors, 
this relationship becomes hyperplanar. By examining 3-dimensional representations of the data, 
a researcher can gain greater insight into the data. The recent report of the APA Task Force on 
Statistical Inference, published in the August, 1999 issue of the American Psychologist, emphasized 
the value and importance of using graphics to understand and communicate data dynamics. 




3 



Regression in 3D 3 



Three-dimensional Modeling in Linear Regression 

The influence of one or more factors on the outcome of a specific event or circumstance 
is an area of interest to many researchers. Extent of these influences can be investigated using 
linear regression. Linear regression can be used to explain the results of a study. An example 
would be if a committee responsible for the selection of incoming medical students tried to 
determine the influence of the previous grades, interview, and scores on the medical college 
aptitude test (MCAT) on the overall ranking of the candidates. Linear regression can also be 
used to predict. For example, if the same committee were interested in determining the how the 
ranking of the candidates would predict the performance of the incoming class in the first year of 
medical school, they could access the information fi'om the previous class, along with their 
grades in their first year of medical school. By using regression, the committee could predict 
fi'om the information fi'om the previous class how the incoming class should perform. 

Geneticist Francis Galton noted that the heights of sons had a tendency to regress toward 
the mean height of the population as compared to the height of their fathers. Sons of a tall man 
are more likely to be shorter, or closer to the mean height, while sons of a short man are more 
likely to be taller. Galton developed regression analysis in order to study this effect (Galton, 
1889). Linear regression produces weights that are used on the predictor variables in order to 
produce optimal values for the dependent or outcome variable, minimizing error. The weight or 
regression coefficient is determined by dividing the sum of squares of the cross product by the 
sum of squares of the predictor variable (Hinkle, Wiersma, & Jurs, 1998: see also Christensen, 
1996; Hocking, 1985). The formula used for simple linear regression is y = a + bx + e, where y is 
the predicted outcome variable, “e” is the error term or y -y, “a” is the y-intercept, and “b” is the 
regression coefficient (See Figure 1). This formula describes a regression line that has the 
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smallest sum of squares for error (SSE). Both y and e are latent or synthetic variables, in that 
they are created as a result of regression, and may not bear any resemblance to the actual data. 
This formula can be converted into standard score form by setting “a” = 0 and b = P(Sy/Sx). In 
standard score form, P is the regression coefficient or beta coefficient. Values for P can range 
from -1 to 1. With two predictor variables, the formula in raw score form is y = a + biXi + b 2 X 2 + 
e, and is = PiZi + P 2 Z 2 in standard score form. Instead of being represented as a line, these 
data can be represented as a plane. Berk (1998) argued that plots of regression of two or more 
predictors in two dimensions do not provide comprehensive insight into the data set. In fact. 
Berk points out that three-dimensional plots can be used to represent multi-dimensional data, by 
“stacking” or layering additional variables along the X 2 axis. 

In multiple regression, the “fit” of the plane to the data is often described by the multiple 
correlation coefficient, R, which can be determined using p. The multiple correlation coefficient 
is a Pearson product-moment correlation between the outcome variable, y, and the predicted 
outcome variable, y. In the case of one predictor variable, R is equal to rxy. In case of two or 
more uncorrelated predictor variables, R is equal to the sum of rxiy, rx 2 y, etc. By taking the square 
root of the sum of the products of each P and its corresponding correlation between x, and y, R 
can be calculated (Hinkle, Wiersma & Jurs, 1998). 

Another valuable tool for examining the relationship between predictor variables and the 
dependent variable are structure coefficients. These are zero-order correlations between the 
predictor variables and the synthetic variable divided by R (Pedhazur, 1982). Mathematically, 
this partitions the influence of each predictor variable on y as seen in the equation rs = rxy/R. As 
explained by Thompson and Borrello (1985), data may be misinterpreted if structure coefficients 
are not examined. Additionally, Thompson points out that examination of structure coefficients 
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can provide information about the collinearity of the predictor variables and the presence of 
suppressor variables (Thompson, 1992). 

This paper has two purposes: (a) to represent linear regression in a three-dimensional 
form to provide visual insight into the influences of various factors on y, and (b) to investigate 
the value of including structme coefficients in the graphs. A data set found in Table 1 will be 
used periodically to generate the graphics. When values for specific coefficients are altered 
artificially in order to examine their effects, the data set will not be used and a dot plot will not 
be included in the graph. Graphs will be limited to two independent (predictor) variables and 
one dependent (outcome) variable. Graphs will not be standardized for better visualization of the 
data away from the x-, y-, and z-axes. More than two predictors and the value of 
multidimensional graphs will be investigated in a another study. Analysis was performed using 
SYSTAT 8.0. Graphs were generated by SigmaPlot 5.0. . 

The use of computer graphics to explore and understand data is consistent with the recently- 
released report of the APA Task Force on Statistical Inference (Wilkinson & The APA Task Force 
on Statistical Inference, 1999). As the Task Force emphasized. 

As soon as you have collected your data, before you compute any statistics, 
look at your data . Data screening is not data snooping. It is not an opportunity 
to discard data or change values to favor your hypotheses. However, if you 
assess hypotheses without examining your data, you risk publishing 
nonsense.... Graphical inspection of data offers an excellent possibihty for 
detecting serious compromises to data integrity. The reason is simple: 

Graphics broadcast; statistics narrowcast. (p. 597, emphasis in original) 
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Certainly such admonitions are not new (Tukey, 1977). And several resources are available 
for researchers seeking guidance on the use of graphics to explore and understand data (cf 
Chambers, Cleveland, Kleiner & Tukey, 1983; Cleveland, 1995; Wilkinson, 1999). But 
microcomputer software has been improved markedly in recent years as regards graphic 
capabilities, and so now these apphcations are readily available to applied researchers. 

Linear Regression Using Two Predictor Variables 
General Graphic Presentation 

Figure 2 attempts to clarify the direction of the influence of each predictor on the 
outcome by holding one predictor constant while altering the other. Initially, bi is sequentially 
changed from -1 to 1, while b 2 is held at 0.5. In Figure 3, bi is held at 0.5 while b 2 is 
sequentially changed from -1 to 1 . The scatterplot and values for the y-axis have been removed 
for the sake of clarity. Note in both examples, the plane rotates on the mean for xi (20) and X 2 
(20), depending upon whether bi or b 2 is being altered. Additionally, at near-zero values for bi 
and b 2 , the plane settles near the mean for y (20). This seems intuitive in that if Xi and/or X 2 are 
useless predictors of y, then the best predictor for y is the mean of y (Thompson, 1992). 

From this perspective, the first predictor variable changes the slope of the plane from left 
to right. Positive values for bi will result in a rise in the plane from left to right, while negative 
values result in left-to-right declination. In this perspective, the second predictor variable 
influences the slope from the front to the back. Positive values for b 2 will result in a rise in the 
plane from front to back, while negative values result in a declination of the plane in the same 
direction. Rotating the image in any direction can easily confuse this, as seen in Figiue 4. In this 
figure, the regression plot is rotated 45° in each sequential frame. It is necessary to note the 
direction of the scale on each labeled axis when looking at graphics in three dimensions. 
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Using Uncorrelated Predictors 

Figure 5 demonstrates the relationship between two predictors and one outcome using the 
equation y = 23.248 + -0.379xi+ 0.217x2 + e as derived from using Predictor 1 and Predictor 2 as 
seen in Figure 4. Table 2 shows the Pearson correlation matrix for all variables, while Table 3 
contains the regression data for Predictor 1 and Predictor 2 versus Outcome. The value for hi is 
the same as in Figure 1, and the addition of the second set of values for Predictor 2 demonstrates 
the relationship between Predictor 2 and the outcome independent of the influence of Predictor 1 
because xi is uncorrelated with X 2 . 

In this case, R is equal to the sum of r xiy and r x 2 y ■ Looking at the regression sums of 
squares (SOSr ) of this data, for Predictor 1 and the outcome variable, the SOSr is equal to 2.736. 
The SOSr for Predictor 2 is 0.895. Totaling these two values results in the same SOSr obtained 
by regressing Predictor 1 and Predictor 2 against the outcome variable. Because the two 
predictor variables are uncorrelated, they do not share any of the sums of squares for y. 

Therefore, any contribution of individual predictors toward explaining y is additive. 

Using Correlated Predictors 

In the previous data set, predictor variables were uncorrelated. However, correlation 
between predictor variables influences the graphic. Using Predictor 1 and Predictor 4, which has 
approximately the same correlation with y (-0.379 for xi and 0.290 for X 2 ), the regression 
equation becomes y = 22.936 + -0.3 14xi + 0.167x2 + e with the correlation of xi with X2 = -0.393 
as seen in Figure 6. Note that the declination of the plane on the xi axis is lessened when the 
predictors are correlated. As each predictor variable only has a limited sums of squares that it 
can use to explain y, any sharing of sums of squares through correlation with other predictor 
variables will limit its influence on y. This is seen using the same comparison of SOSr as losted 
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in Table 4. The SOSr for predictor 1 is 2.736 and SOSr for Predictor 4 is 1 .601 . However, when 
Predictors 1 and 4 are regressed together with y, the SOSr is 3.1 84, indicating that some of the 
sums of squares explained by the predictors overlap. The correlation between Predictor 1 and 
Predictor 4 is -0.393. 

Regressing Predictor 3 and Predictor 5 with y provides insight into to the effects of 
positively correlated independent variables (r = 0.421, Figure 7). For Predictor 3 regressed with 
y, the SOSr is 4.413, while the SOSr for Predictor 5 with y is 3.666 (from Table 5). The SOSr 
for both predictors with y is 5.703. Regardless of the sign of the correlation coefficient, 
collinearity reduces the usefulness of an individual predictor variable when using multiple 
predictors in most cases. Figure 8 demonstrates is an overlay of Figure 7 with the collinearity of 
the two predictors removed. 

Value of Structural Equations 

Because of the influence of collinearity of predictor variables on linear regression, proper 
interpretation of data requires more information (Thompson, 1992; Thompson & Borrello, 1985). 
In Figure 9, Predictor 4 is regressed with Predictor 6 against the outcome resulting in bi = 0.664, 
b 2 = 0.592 and R^ = 0.296 (adjusted R^ = 0.213). The Pearson correlation coefficients for these 
data sets are rp 4 ,y = 0.290, rpg.y = 0.174, and rp 4 ,pg = -0.630. This is contradictory to the previous 
discussion in that the resulting R^ increases when these two correlated predictors are used. The 
SOSr for Predictor 4 by itself is 1 .601, while the SOSr for Predictor 6 is 0.576. The SOSr 
regressing both predictors is 5.619 as seen in Table 6. Figure 10 show an overlay of Figure 9 
with the red grid representing the weights for Predictors 4 and 6 with the correlation removed. 
This is due to the presence of a suppressor variable. By definition, a suppressor variable is a 
variable that increases R^ with its addition and is not or weakly correlated with the dependent 
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variable Thompson, 1992). In this case, Predictor 6 is modestly correlated with y (r = 0.174) 
while Predictor 4 is more correlated (r = 0.290). By examining the structure coefficients (rs) of 
these two variables, it can be determined which explains the most of y. The structure 
coefficients can be calculated by dividing by R. The correlation between Predictor 4 and y 
(rp 4 .v) is 0.534, while the correlation between Predictor 6 and y (i je.v) is 0.320. The structure 
coefficient for Predictor 4 (rsp 4 ) is 0.9816, while rspe is 0.5882. Predictor 4 occupies 98% of the 
sums of squares explained and Predictor 6 occupies 59%. See Figure 1 1 for the addition of the rs 
for this data set. Another method by which the presence of suppressor variable can be 
determined is by regressing the predictors with and without the suspected suppressor. 

Regressing Predictors 4 and 5 with and without Predictor 6 yields an R without Predictor 6 of 
0.491, and an R with Predictor 6 of 0.564 as seen in Table 7. The r for Predictor 4 increased 
from 0.223 with out Predictor 6 to 0.540 with Predictor 6. The correlation coefficient for 
Predictor 5 decreased from 0.402 to 0.191 when Predictor 6 was added. All three predictors are 
correlated, as seen in Table 1. 

Linear regression when used as a tool for prediction or explanation is usually indicative 
of extreme interest in the data. However, b weights, P weights and correlation coefficients offer 
only incomplete insight into the nature of the predictors. Viewing the data set graphically and 
including the “best fit” plane allows the investigator to see how much collinearity is affecting the 
data. In general, collinearity reduces the slope of the plane, due to sharing of the sums of squares 
by the predictor variables. However, when a suppressor variable is present, collinearity increases 
the slope of the plane. Normally, this would not be apparent unless the investigator chose to 
overlay the regression graph with a graph of the correlation coefficients. By adding the structure 
coefficients, the suppressor variable can be identified as it should be uncorrelated with the 
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outcome variable, but be credited with having a positive effect on the prediction validity of the 
model. Three-dimensional representation of data sets helps further elucidate the data by helping 
the researcher visualize the data as well as the values of the various coefficients. 
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Figure Caption 

Figure 1. Simple linear regression with b = -0.379. 

Figure 2. Linear regression with two predictor variables varying hi and holding \>2 constant at 
0.5. In graph A, hi = -1.0 and \>2 = 0.5. In graph B, hi = -0.75 and \>2 = 0.5. In graph C, hi = -0.5 
and b2 = 0.5. In graph D, hi = -0.25 and b2 = 0.5. In graph E, hi = 0.0 and b2 = 0.5. In graph F, 
b] = 0.25 and b2 = 0.5. In graph G, bi = 0.5 and b2 = 0.5. In graph H, bi = 0.75 and b2 = 0.5. In 
graph I, bi = 1.0 and b2 = 0.5. 

Figure 3. Linear regression with two predictor variables holding bi constant at 0.5 and varying 
b2. In graph A, bi = 0.5 and b2 = -1.0. In graph B, bi = 0.5 and b2 = -0.75. In graph C, bi = 0.5 
and b2 = -0.5. In graph D, bi = 0.5 and b2 = -0.25. In graph E, bi = 0.5 and b2 = 0.0. In graph F, 
bi = 0.5 and b2 = 0.25. In graph G, bi = 0.5 and b2 = 0.5. In graph H, bi = 0.5 and b2 = 0.75. In 
graph I, b] = 0.5 and b2 = 1.0. 

Figure 4. Linear regression with two predictor variables with sequential 45° rotation, starting at 
0° and ending at 315°. 

Figure 5. Linear regression with two predictor variables with bi = -0.379 and b2 = 0.217. 

Figure 6. Linear regression with two predictor variables with bi = -0.3 14 and b2 = 0. 1 67 (top). 
Side-by-side comparison of plots of previous predictors (bottom right) and current predictors 
(bottom left). 

Figure 7. Linear regression with two positively correlated predictors with bi = 0.361 and 62 = 
0.287. 

Figure 8. Linear regression with two positively correlated predictors with bi = 0.361 and 62 == 
0.287. Red grid represents linear regression with two positively correlated predictors with bi = 
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0.482 and b2 = 0.439. 
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Figure 9. Linear regression with bi = 0.664 and 62 = 0.592 as derived from predictor variables 4 
and 6 . 

Figure 10. Linear regression with bi = 0.664 and hi = 0.592 as derived from predictor variables 4 
and 6 . Red grid represents overlay of predictor variables 4 and 6 without collinearity, bi = 0.290 
and b 2 = 0.174. 

Figure 11. Linear regression with bi = 0.664 and 62 = 0.592 as derived from predictor variables 4 
and 6 . Red grid represents overlay of predictor variables 4 and 6 without collinearity, bi = 0.290 
and b 2 = 0.174. Blue grid represents an overlay of the structure coefficient for Predictor 4 and 



Predictor 6 , with bi = 0.9816 and 62 = 05882. 
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Simple Linear Regression 
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Two Predictors 
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Positively Correlated Predictors 
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Positively Correlated Predictors 





23 



Regression in 3D 23 



Linear Regression with Two Correlated Variables 
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Data Set Used for Linear Regression 



Outcome 


Predictor 1 


Predictor 2 


Predictor 3 


Predictor 4 


Predictor 5 


Predictor 6 


19.553 


21.61 


19.338 


19.162 


19.718 


19.488 


20.24 


20.094 


20.537 


21.545 


20.576 


19.64 


19.925 


21.286 


20.799 


19.732 


19.89 


20.386 


19.662 


19.889 


20.641 


20.778 


19.195 


18.786 


19.646 


20.297 


21.399 


21.116 


20.296 


18.927 


19.338 


20.579 


19.924 


19.732 


19.904 


21.42 


18.927 


20.662 


20.598 


20.704 


20.303 


20.223 


19.582 


19.195 


21.214 


18.595 


19.35 


18.549 


19.095 


20.345 


19.732 


20.11 


19.087 


21.979 


19.566 


18.004 


19.988 


20.537 


18.455 


20.386 


18.923 


19.148 


21.652 


20.86 


21.61 


20.662 


20.806 


20.068 


19.481 


19.781 


19.753 


18.927 


20.662 


19.768 


21.384 


19.325 


18.4 


20.491 


19.195 


21.214 


21.681 


19.026 


20.357 


19.841 


18.415 


19.732 


20.11 


18.873 


19.657 


20.294 


19.378 


19.474 


20.537 


18.455 


21.746 


18.945 


21.679 


20.997 


19.506 


21.61 


20.662 


19.755 


20.467 


20.51 


20.224 


17.166 


21.61 


19.338 


18.393 


19.058 


17.365 


19.21 


20.48 


20.537 


21.545 


20.857 


18.217 


20.556 


21.488 


21.158 


19.732 


19.89 


20.76 


20.537 


20.344 


19.275 


19.067 


19.195 


18.786 


19.834 


20.541 


21.022 


20.03 


20.778 


18.927 


19.338 


18.512 


21.904 


21.07 


19.216 


Avg 


20 


20 


20 


20 


20 


20 


20 


sd 


1 


1 


1 


1 


1 


1 


1 



Table 2 



Pearson correlation matrix 





Outcome 


Predictor 

1 


Predictor 

2 


Predictor 

3 


Predictor 

4 


Predictor 

5 


Predictor 

6 


Outcome 


1.000 














Predictor 1 


-0.379 


1.000 












Predictor 2 


0.217 


0.000 


1.000 










Predictor 3 


0.482 


-0.005 


0.125 


1.000 








Predictor 4 


0.290 


-0.393 


-0.080 


-0.376 


1.000 






Predictor 5 


0.439 


-0.312 


-0.186 


0.421 


0.167 


1.000 




Predictor 6 


0.174 


0.269 


-0.144 


0.509 


-0.630 


0.354 


1.000 





Table 3 



Regression of Predictor 1 and Predictor 2 



Dep Var: Outcome N: 20 Multiple R: 0.437 Squared multiple R: 0.191 
Adjusted squared multiple R: 0.096 Standard error of estimate: 0.951 



Effect 


Coefficient 


Std Error 


Std Coef Tolerance 


t 


P(2 Tail) 


CONSTANT 


23.248 


6.174 


0.000 




3.765 


0.002 


Predictor 1 


-0.379 


0.218 


-0.379 


1.000 


-1.739 


0.100 


Predictor 2 


0.217 


0.218 


0.217 


1.000 


0.995 


0.334 




Analysis of Variance 










Source 


Sum-of-Squares df 




Mean-Square 


F-ratio 


P 


Regression 


3.631 


2 




1.815 


2.007 


0.165 


Residual 


15.373 


17 




0.904 
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Table 4 



Regression of Predictor 1 and Predictor 4 



Dep Var: Outcome N: 20 Multiple R: 0.409 Squared multiple R; 0.168 
Adjusted squared multiple R: 0.070 Standard error of estimate: 0.965 



Effect 


Coefficient 


Std Error 


StdCoef Tolerance 


t 


P(2 Tail) 


CONSTANT 


22.936 


80.35 




0.000 




2.854 


0.011 


Predictor 1 


-0.314 


0.241 




-0.314 


0.846 


-1.304 


0.210 


Predictor 4 


0.167 


0.241 




0.167 


0.845 


0.694 


0.497 




Analysis of Variance 










Source 


Sum-of-Squares 


df 




Mean-Square 


F-ratio 


P 


Regression 


3.184 




2 




1.592 


1.711 


0.210 


Residual 


15.820 




17 




0.931 
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Table 5 



Regression of Predictor 3 and Predictor 5 



Dep Var: Outcome N: 20 Multiple R: 0.548 Squared multiple R: 0.300 
Adjusted squared multiple R: 0.218 Standard error of estimate: 0.885 



Effect 


Coefficient 


Std Error 


Std Coef Tolerance 


t 


P(2 Tail) 


CONSTANT 


7.037 


4.819 


0.000 




1.460 


0.162 


Predictor 3 


0.361 


0.224 


0.361 


0.823 


1.613 


0.125 


Predictor 5 


0.287 


0.224 


0.287 


0.823 


1.284 


0.216 




Analysis of Variance 










Source 


Sum-of-Squares df 




Mean-Square 


F-ratio 


P 


Regression 


5.703 


2 




2.851 


3.644 


0.048 


Residual 


13.301 


17 




0.782 
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Table 6 



Regression of Predictor 4 and Predictor 6 



Dep Var: Outcome N: 20 Multiple R: 0.544 Squared multiple R: 0.296 
Adjusted squared multiple R: 0.213 Standard error of estimate: 0.887 



Effect 


Coefficient 


Std Error 


Std Coef Tolerance 


t 


P(2 Tail) 


CONSTANT 


-5.114 


9.470 


0.000 




-0.540 


0.596 


Predictor 4 


0.664 


0.262 


0.663 


0.603 


2.531 


0.022 


Predictor 6 


0.592 


0.262 


0.592 


0.603 


2.259 


0.037 




Analysis of Variance 










Source 


Sum-of-Squares df 




Mean-Square 


F-ratio 


P 


Regression 


5.619 


2 




2.809 


3.568 


0.051 


Residual 


13.385 


17 




0.787 
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Table 7 



Regression of Predictor 4. Predictor 5 and Predictor 6 



Dep Var: Outcome N: 20 Multiple R: 0.564 Squared multiple R: 0.318 
Adjusted squared multiple R: 0.190 Standard error of estimate: 0.900 



Effect 


Coefficient 


Std Error 


Std Coef Tolerance 


t 


P(2 Tail) 


CONSTANT 


-3.552 


9.840 


0.000 




-0.361 


0.723 


Predictor 4 


0.540 


0.315 


0.540 


0.429 


1.713 


0.106 


Predictor 5 


0.191 


0.262 


0.191 


0.622 


0.729 


0.477 


Predictor 6 


0.447 


0.332 


0.447 


0.386 


1.344 


0.198 




Analysis of Variance 










Source 


Sum-of-Squares df 




Mean-Square 


F-ratio 


P 


Regression 


6.049 


3 




2.016 


2.490 


0.097 


Residual 


12.955 


16 




0.810 
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